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ABSTRACT 


The  Department  of  Defense’s  future  vision  for  Network  Centric  Operations  (NCO) 
will  increase  combat  power  by  networking  relevant  entities  across  the  battlefield.  This 
will  result  in  highly  complex  mission  scenarios  in  which  the  operator’s  workload  will  be 
easily  overloaded  if  the  system  is  not  designed  to  support  the  mission  requirements. 
New  technologies  for  these  complex  command  and  control  environments  are  currently 
being  developed.  However,  little  has  been  done  to  evaluate  the  adequacy  of  a  particular 
technology  for  specific  sets  of  mission  requirements.  There  is  neither  a  standard 
methodology  to  evaluate  these  new  technologies,  nor  a  research  environment  to  test 
these  technologies  under  realistic  assumptions.  This  paper  will  introduce  a  new 
approach  to  evaluate  these  technologies  and  determine  whether  they  can  transition  into 
practical  applications  for  the  Navy,  and  under  which  limitations. 
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I.  INTRODUCTION 


The  Department  of  Defense’s  future  vision  for  Network  Centric  Operations  (NCO) 
is  intended  to  increase  combat  control  by  networking  relevant  entities  across  a 
battlefield  [1].  This  new  vision  implies  large  amounts  of  information  sharing  and 
collaboration  across  different  entities.  An  example  of  a  futuristic  NCO  scenario  is  one  in 
which  a  group  of  heterogeneous  Unmanned  Vehicles  (UVs)  are  supervised  by  a  single 
operator  using  NCO  technology.  In  this  type  of  complex  command  and  control  (C2) 
scenario,  UV  operators  will  be  subjected  to  vast  amounts  of  information  as  compared  to 
today’s  command  and  control  scenarios.  Therefore,  this  vision  brings  with  it  a  new 
problem  that  must  be  addressed:  How  to  maintain  an  adequate  workload  to  avoid 
information  overload  and  resulting  loss  of  situation  awareness.  Currently,  C2 
technologies  that  allow  the  operator  to  control  multiple  UVs  in  a  NCO  scenario  are 
rapidly  increasing.  The  development  of  these  new  C2  technologies  generates  the 
tendency  to  exponentially  increase  the  ratio  of  UVs  to  operators.  However,  if  systems 
are  inadequately  designed  or  are  used  beyond  their  design  capabilities,  they  will  not 
adequately  control  for  increased  workload,  which  in  turn  will  cause  the  operator  to 
become  overloaded  and  lose  situation  awareness.  It  is  critical  that  military  decision 
makers  develop  predictive  models  of  human  and  system  performance  to  evaluate  the 
adequacy  of  a  system’s  design  to  satisfy  specific  mission  requirements. 


II.  BACKGROUND 

Mental  workload  is  a  limiting  factor  in  deciding  how  many  UVs  an  operator  can 
control  or  supervise.  In  the  case  of  one  operator  supervising  multiple  vehicles,  the 
operator’s  workload  is  measured  by  the  effort  required  to  supervise  each  vehicle  and 
the  overall  task.  The  effort  required  to  supervise  an  individual  UV  in  a  team  depends  on 
the  efficiency  of  the  system  to  reduce  workload  and  increase  situation  awareness. 
Moreover,  workload  also  depends  on  the  complexity  of  the  mission  scenario.  Some  of 
the  characteristics  of  a  complex  mission  scenario  as  defined  by  military  standards 
include:  mission  time  constraints,  precision  constrains,  repeatability  in  tasks  (i.e., 
navigation,  manipulations,  etc.),  level  of  collaboration  required,  concurrence  and 
synchronization  of  events  and  behaviors,  resource  management  (i.e.,  power, 
bandwidth,  ammunition),  rules  of  engagement,  adversaries,  and  knowledge 
requirements  [2],  The  degree  to  which  these  characteristics  are  required  also  define 
workload.  Consequently,  if  the  system  is  not  designed  to  achieve  specific  types  of 
requirements,  then  when  it  is  tested  for  those  requirements  the  system  may  not  perform 
them  adequately. 
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Previous  attempts  to  model  operator  capacity  were  developed  to  display 
temporal  constraints  associated  with  the  system.  The  complexity  of  these  measures 
progressed  from  measuring  operator  capacity  in  homogenous  UVs  controlled  by  one 
operator  [3-7],  to  scenarios  in  which  teams  of  heterogeneous  UVs  are  supervised  by 
one  operator  [8],  The  first  equation  developed  to  predict  operator  capacity  in 
homogenous  UVs  suggested  that  the  operator  capacity  is  a  function  of  the  Neglect  Time 
(NT),  or  the  time  the  UV  operates  independently,  and  Interaction  Time  (IT),  or  the  time 
the  operator  is  busy  interacting,  monitoring,  and  making  decisions  with  the  system  [3], 
Critics  of  this  method  suggested  that  the  equation  lacked  two  critical  considerations:  1 ) 
the  importance  of  including  Wait  Times  (WTs)  caused  by  human-vehicle  interaction,  and 
2)  how  to  link  this  equation  to  measure  effective  performance  [6],  Hence,  WTs  were 
added  to  the  equation  to  account  for  the  times  the  UV  has  to  perform  in  a  degraded 
state  because  the  operator  is  not  able  to  attend  to  it  or  is  not  aware  of  a  new  incoming 
event.  Three  WTs  were  identified:  Wait  Times  due  to  Interaction  (WTI),  Wait  Times  due 
to  Loss  of  Situation  Awareness  (WTSA),  and  Wait  Times  due  to  Queue  (WTQ). 

Carl  Nehme  from  the  Massachusetts  Institute  of  Technology  (MIT)  developed  the 
Multiple  Unmanned  Vehicles  Discrete  Event  Simulation  (MUV-DES).  He  attempted  to 
create  a  link  to  performance  by  using  proxies  to  measure  workload  and  situation 
awareness  [8],  In  this  model,  the  researcher  intended  to  model  heterogeneity  in  UV 
systems  in  order  to  evaluate  the  system’s  design.  The  human  was  modeled  as  a  server 
attending  to  vehicle-generated  tasks  -  both  exogenous  and  endogenous  tasks  -  as 
defined  by  their  arrival  and  service  processes.  The  concept  of  utilization  was  introduced 
as  a  proxy  for  measuring  mental  workload.  Utilization  Time  (UT)  refers  to  the 
percentage  of  time  the  operator  is  busy.  The  concept  of  WTSA  was  used  as  a  proxy  to 
measure  Situation  Awareness.  The  UT  and  WTSA  measures  were  computed  as  a  type 
of  aggregate  effect  of  inefficiencies  in  information  processing  rather  than  being 
computed  as  individual  measures  of  workload  and  situation  awareness.  The  author  of 
this  model  suggested  that  many  other  sources  of  cognitive  inefficiencies,  besides  these 
two  proxies,  are  manifested  through  cognitive  delays.  He  emphasized  that  measures  of 
UT  and  WTSA  are  extremely  critical  to  determine  supervisory  performance  and 
suggested  that  better  methodologies  to  measure  these  variables  need  to  be  developed. 

III.  PROJECT  GOALS 

This  study  aims  to  develop  a  model  of  operator  capacity  in  a  complex  mission 
scenario  that  converges  all  previous  research  in  the  area  to  create  a  more 
comprehensive  model  of  operator  capacity.  This  comprehensive  model  is  intended  to  fill 
in  the  gaps  of  current  research  by  introducing  new  variables  and  relationships  to 
previous  models.  The  model  will  be  constructed  in  a  way  so  prior  knowledge  about  the 
relationship  between  variables  will  serve  to  better  predict  missing  data,  such  as 
workload  and  situation  awareness.  Moreover,  the  model  will  be  structured  in  a  way  that 
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will  make  it  easy  to  determine  which  areas  in  the  system  design  need  improvement.  The 
ultimate  goal  of  this  study  is  to  develop  a  decision-making  tool  that  will  serve  to  evaluate 
and  determine  the  effectiveness  and  limitations  of  a  particular  NCO  technology  in  a 
complex  mission  scenario. 


IV.  METHODOLOGY 


A.  Approach 

The  approach  taken  by  this  research  study  was  to  model  the  decision-making 
process  required  to  decide  whether  to  increase  a  particular  team  size.  This  approach 
was  taken  in  order  to  present  decision  makers  with  a  decision-support  tool  that  will 
ensure  that  knowledgeable  decisions  are  made  in  regards  to  the  adequacy  of  a  given 
team  size  with  a  particular  NCO  technology.  Modeling  the  decision-making  process,  as 
opposed  to  the  environment,  allows  for  more  knowledgeable  decisions  because  not  only 
are  the  most  important  factors  in  the  decision  taken  into  account,  but  optimization  of  the 
recommended  decision’s  outcome  is  also  possible.  This  approach  provides  adequate 
information  to  the  user  to  make  a  decision.  And  while  the  model  is  based  on  answering 
this  particular  question,  the  nature  of  the  situation  is  manifested  in  the  model,  thus 
allowing  users  to  draw  more  conclusions  than  only  the  adequacy  of  the  team  size. 

B.  The  Model 

/.  Model  Overview 

A  decision  network  was  developed  to  model  the  decision-making  process 
required  to  decide  whether  to  increase  a  given  team  size  with  the  selected  NCO 
technology.  Netica  Software  [9]  was  used  to  develop  a  Bayesian  decision  network  that 
incorporates  quantitative  and  qualitative  information  about  the  model.  This  software  was 
chosen  mainly  because  it  can  accommodate  missing  or  incomplete  data.  Using  Netica 
allows  researchers  to  compute  unobservable  variables  (i.e. ,  missing  data)  based  on 
measures  that  are  observed  (i.e.,  prior  knowledge). 

A  decision  network  consists  of  nature,  decision,  and  utility  nodes.  Nature  nodes 
represent  variables  over  which  the  decision  maker  has  no  control.  Decision  nodes 
represent  variables  over  which  the  decision  maker  can  make  a  decision  (see  blue 
nodes  in  Fig.  1).  Utility  nodes  represent  a  measure  of  value,  or  the  decision  maker’s 
preferences  for  the  states  of  the  variables  in  the  model  (see  pink  nodes  in  Fig.  1 ).  In  this 
type  of  network,  the  outcome  of  a  decision  node  is  maximized  by  finding  a  configuration 
of  the  various  states  of  the  sets  of  variables  that  maximize  the  values  of  the  utility  node. 
Therefore,  based  on  a  series  of  requirements,  or  utility  values,  a  decision  network 
provides  the  user  with  the  correct  decision.  Additionally,  the  arrows  in  the  model 
represent  reasoning  relationships  and  are  detailed  in  the  conditional  probability  tables 
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(CPTs)  of  the  nature  and  utility  nodes.  In  the  CPT,  the  distribution  of  each  node  will  be 
determined  a  priori  based  on  the  relationships  specified  in  each  conditional  probability 
table. 

/'/'.  Model  Assumptions 

This  model  makes  several  assumptions.  First,  the  type  of  UV  system  addressed 
by  this  model  is  one  in  which  a  single  human  operator  is  responsible  for  supervising  a 
team  of  heterogeneous  UVs.  The  human  operator  is  assumed  to  act  in  a  supervisory 
control  style,  interacting  with  the  system  at  discrete  points  in  time  (i.e.,  there  is  no 
manual  control).  Second,  in  this  model,  the  human  operator  is  responsible  for 
supervising  a  team  of  heterogeneous  UVs  defending  an  oil  platform  from  potential 
enemies.  Third,  the  human  operator  could  be  situated  in  a  ground-based,  sea-based,  or 
airborne  control  station.  Fourth,  the  model  was  built  in  a  way  such  that  decision  makers 
will  use  this  model  to  help  them  decide  if  a  particular  technology  is  adequate  for  specific 
mission  requirements.  Finally,  the  model  assumes  that  the  decision  making  process 
required  to  make  this  decision  is  hierarchical;  therefore,  later  decisions  are  based  on 
earlier  ones.  The  model  captures  attributes  from  the  Operator  Performance  Model,  the 
System  Performance  Model,  and  the  Operator  Capacity  Model  as  shown  in  Figure  1 . 


Operator  Attention 
Allocation  Strategies 

Operator  DM 
Efficiency 


Workload 


r  ' 

Operator 

System 

Performance 

Model 

k.  A 

Performance 

Model 

k.  A 

Operator 

Capacity 


Usability 

Automation  Level 
Algorithms  Efficiency 


Adequate 
Team  Size 


Fig.  1.  A  high  level  representation  on  the  attributes  the  model  captures.  Notice  that  variables  of  interest  in  Operator 
Performance  Model  are  Operator  Attention  Allocation  Strategies  and  Operator  Decision  Making  Efficiency,  while  in  the 
System  Performance  model  are  Usability,  Automation  Level  and  Algorithm  Efficiency.  The  output  of  the  operator  capacity 
model  is  to  determine  an  adequate  team  size. 
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iii.  Model  Description 

The  model  is  based  on  three  major  areas  of  relevance  for  the  decision  to 
increase  the  team  size:  system  performance,  operator  performance,  and  cognitive 
workload  (See  Fig.  2).  These  areas  of  relevance  are  represented  in  the  model  as  sub¬ 
models;  each  of  them  contains  one  or  more  decision  nodes  that  correspond  to  the 
decisions  that  must  be  made  by  the  operator  in  each  area  to  ensure  that  they  are 
working  adequately.  The  order  in  which  the  decision  nodes  have  been  organized 
represents  the  way  in  which  decisions  should  be  made  (see  blue  nodes  on  Fig.  1).  The 
model  represents  a  sequence  of  decisions  in  which  later  decisions  depend  on  the 
results  of  earlier  ones.  In  this  model,  the  last  decision  is  shown  at  the  end  of  the 
sequence.  The  last  decision  determines  whether  the  team  size  should  be  increased. 

The  first  sub-model,  system  performance,  includes  three  decision  nodes  with  the 
followings  decisions:  1 )  Is  the  interface  effective?  2)  Does  the  system  have  an  adequate 
level  of  automation?  3)  Are  the  system  algorithms  efficient  for  the  task?  These  three 
decisions  were  included  in  this  sub-model  because  they  represent  areas  that  are 
important  to  ensure  good  system  performance.  Some  of  the  utility  nodes  for  each  of 
these  decision  nodes  were  identified  from  the  literature,  while  some  others  were 
included  to  ensure  that  specific  mission  requirements  are  satisfied.  For  example,  if  the 
system  has  good  interface  usability,  the  situation  awareness  of  the  operator  will  be  high. 
Moreover,  if  the  situation  awareness  is  high,  the  system’s  automation  level  must  be 
somehow  effective  to  avoid  loss  of  situation  awareness  and/or  complacency.  Then,  to 
ensure  that  the  mission  requirements  are  satisfied,  the  algorithms  used  must  be  working 
efficiently  toward  achieving  the  mission  goal.  This  efficiency  is  measured  by  the  number 
of  times  the  operator  reassigns  a  mission  that  was  previously  assigned  by  the  system, 
with  a  lower  number  signaling  higher  efficiency.  Note  that  algorithm  efficiency  is  defined 
in  this  model  only  as  a  result  of  the  operator’s  perceived  trustworthiness  of  the  system. 
If  the  system  is  not  perceived  as  trustworthy,  then  the  operator  will  tend  to  override  the 
system  frequently  and  the  algorithm  efficiency  will  be  low. 

The  second  sub-model,  operator  performance,  needs  to  ensure  that  the  operator 
performs  effectively  with  the  system  being  evaluated,  as  more  UVs  are  introduced  to  the 
team,  and  the  mission  scenario  becomes  more  complex.  Since  this  is  a  supervisory 
control  environment,  operator  performance  is  defined  in  terms  of  the  operator’s  decision 
making.  There  are  two  decisions  (decision  nodes)  that  are  important  to  evaluate 
whether  the  operator’s  performance  is  adequate  for  the  task:  1 )  Is  the  operator’s  task 
management  strategy  efficient?  2)  Is  the  operator’s  decision  making  efficient?  The  first 
decision  is  necessary  to  evaluate  whether  operators  will  efficiently  prioritize  different 
tasks  that  arrive  simultaneously.  The  second  decision  is  necessary  to  evaluate  whether 
the  operator  will  successfully  achieve  the  goals  of  the  mission  (i.e.,  protecting  the  asset 
from  enemy  attack).  Together  these  two  decisions  summarize  what  is  important  to 
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ensure  a  satisfactory  operator  performance.  Please  note  that  by  measuring  task 
management  efficiency,  an  attention  inefficiency  component  is  included  in  this  model. 

Finally,  the  last  sub-model,  cognitive  workload,  includes  the  final  decision  node: 
“Increase  Team?”  For  this  decision,  it  is  important  to  ensure  that  operators  are  not 
overloaded,  but  instead  their  workload  is  adequate  to  successfully  complete  the  mission 
scenario.  This  final  decision  node  is  the  end  of  a  sequence  of  decisions  and  therefore  it 
depends  on  the  outcomes  of  the  previous  decisions  made  in  the  system  performance 
and  operator  performance  sub-models.  Hence,  in  order  to  avoid  cognitive  overload,  not 
only  does  the  system  have  to  efficiently  perform  in  the  mission  scenario,  but  the 
operator  also  has  to  perform  efficiently  to  ensure  that  tasks  are  adequately  managed 
and  do  not  overload  the  operator.  The  cognitive  workload  and  operator  performance 
sub-models  are  strongly  associated.  If  cognitive  workload  is  too  high,  then  the  operator 
performance  will  be  low.  Therefore,  the  more  inadequate  management  and  tactical 
decisions  operators  make,  the  higher  their  workload  will  be. 

System  performance,  operator  performance,  and  cognitive  workload  are  the 
foundation  of  this  model.  Most  of  the  knowledge  about  the  model  relationships  between 
variables  was  acquired  from  a  literature  review.  Variables  such  as  “Information 
Overload”  and  “System  Interruption”  were  included  to  emphasize  the  need  to  evaluate 
these  aspects  of  the  usability  of  the  system  (see  Fig.  2)  in  complex  supervisory  control 
tasks.  These  variables  are  relevant  because  they  contribute  to  design  interfaces, 
especially  in  the  supervisory  control  environment  in  which  large  amounts  of  information, 
and  large  event  queues  can  result  in  information  overload  and  frequent  system 
interruptions. 


Fig.  2.  Decision  network  representing  the  decision  process  involved  in  deciding  whether  to  increase  a  particular  team  size. 
Notice  that  this  picture  displays  the  model  with  no  data.  When  data  are  introduced  into  the  model,  the  system  provides  the  user 
with  a  recommended  course  of  action  that  will  be  displayed  as  a  percentage  (i.e.,  Yes  90%). 
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iv.  Model  Measures 

The  model  allows  for  measurement  of  several  output  variables.  These  variables 
include  those  implemented  in  the  MUV-DES  model,  as  well  as  specific  user-defined 
metrics  that  the  model  allows  to  capture.  Temporal  measures  such  as  UT  and  WT  are 
used  because  they  are  critical  in  a  system  where  the  operator  has  limited  attention 
resources  that  must  be  divided  across  multiple  tasks.  UT  is  used  to  capture  the  effects 
of  alternate  design  variables  on  operator  workload.  Some  researchers  indicate  that 
average  UT  and  WT  can  allow  for  benchmarking  and  comparison  to  be  made  across 
applications  [8,  10].  The  level  of  autonomy  in  the  model  is  captured  through  the  NT.  In 
addition  to  the  basic  metrics  inherently  captured  by  the  MUV-DES  model,  this  model 
also  captures  mission-specific  metrics.  Some  of  the  mission-specific  metrics  include  the 
rate  at  which  tasks  are  successfully  completed,  the  UVs’  health  status  and  the  total  time 
to  complete  the  mission  scenario.  Furthermore,  other  measures  being  captured  by  the 
model  include  Information  Overload,  System  Interruption,  and  Reassignment  Rate. 
These  three  measures  are  important  to  evaluate  the  system  performance.  Information 
Overload  and  System  Interruption  are  shown  to  be  related  to  SA;  therefore,  they  are 
used  to  help  determine  Situation  Awareness  (SA).  For  example,  when  the  operator  is 
overloaded  with  information,  he/she  is  not  able  to  focus  on  what  is  important,  therefore 
vital  SA  is  lost.  Moreover,  when  the  system  is  constantly  interrupting  the  operator  at  any 
point  in  time,  it  drives  the  operator’s  attention  away  from  one  task  to  focus  on  another, 
therefore  affecting  their  SA.  The  system’s  Frequency  of  Reassignment  measure  is  used 
to  evaluate  the  number  of  times  the  operator  overrides  the  system.  Identifying  the 
amount  of  times  the  system  has  been  overridden  will  help  us  determine  how  trustworthy 
the  system  is  for  the  operator.  The  underlying  assumption  is  that  the  more  the  operator 
overrides  the  system,  the  less  reliable  the  algorithm  for  the  system  is.  For  a  list  of  the 
performance  measures  used  in  the  model,  see  Figure  3. 


Performance  Measures 

MUV-DES 

Others 

Wait  Times  due  To  Situation  Awareness(  WTSA) 

X 

Wait  Times  due  to  Queue  (WTQ) 

X 

Wait  Times  due  to  Cognitive  Reorientation  (WTCR) 

X 

Interaction  Times  (IT) 

X 

Neglected  Times  (NT) 

X 

Utilization  Times  (UT) 

X 

Total  Task  Time 

X 

Information  Overload 

X 

System  Interruption 

X 

Target  Elimination  Task-  Success  Rate 

X 

Identification  Task-Success  Rate 

X 

Frequency  of  Reassignment 

X 

UV  Health  Status 

X 

Fig.  3.  Performance  measures  used  in  the  model.  Notice  table  divides  measures  that  are  being 
used  from  the  MUV-DES  model  and  other  measures  that  were  developed  specifically  for  this  model. 
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C.  Model  Validation  and  Data  Collection 


i.  Experimental  Apparatus 

Since  there  is  no  test  bed  available  that  portrays  all  the  complexities  of  a 
futuristic  mission  scenario,  the  Research  Environment  for  Supervisory  Control  of 
Heterogeneous  Unmanned  Vehicles  (RESCHU)  developed  by  MIT  was  acquired  and 
later  modified  to  be  used  as  a  test  bed  in  this  study.  The  RESCHU  simulator  [8]  is  a  test 
bed  that  allows  operators  to  supervise  a  team  of  Unmaned  Aerial  Vehicles  (UAVs)  and 
Unmanned  Underwater  Vehicles  (UUVs)  while  conducting  surveillance  and  identification 
tasks.  This  simulation  was  modified  for  this  study  to  include  the  following  requirements: 
1)  a  complex  mission  scenario  with  an  asset  to  protect  and  multiple  simultaneous 
enemies  to  attack,  2)  a  highly  automated  system  such  as  mission  definition  language 
(MDL)  and  3)  a  highly  heterogeneous  team  that  is  made  of  at  least  three  different  types 
of  UVs.  The  new  version  of  the  simulation  is  called  RESCHU  SP. 

It  is  important  to  mention  that  the  Unmanned  System  technology  selected  as  an 
example  of  a  NCO’s  technology  that  allows  one  operator  to  supervise  multiple  UVs  is 
the  Collaborative  Sensing  Language  (CSL)  developed  at  the  University  of  California, 
Berkeley.  The  CSL  [11]  is  a  high-level  feedback  control  language  for  mobile  sensor 
networks  of  UAVs.  This  system  allows  an  operator  to  concentrate  on  high-level 
decisions,  while  the  system  takes  care  of  low-level  decisions,  like  choosing  which  UV  to 
send  for  a  particular  type  of  task.  A  framework  for  the  CSL  was  designed  to  integrate 
this  technology  into  the  complex  mission  scenario  portrayed  by  the  RESCHU  SP 
simulator.  The  CSL  version  displayed  in  this  simulation  is  only  intended  to  illustrate  one 
way  to  portray  how  this  technology  may  work  in  more  complex  mission  scenarios  and 
with  supervisory  control  of  heterogeneous  UVs  (See  Fig.  4). 

ii.  Vehicle  Types  and  Functions 

The  team  of  UVs  in  the  RESCHU  SP  simulator  is  composed  of  UAVs,  UUVs,  and 
Unmanned  Surface  Vehicles  (USVs).  There  are  two  types  of  UAV,  the  MALE  UAV  and 
the  HALE  UAV;  both  travel  to  areas  of  interest  to  detect  potential  enemies.  When  a  UAV 
detects  a  potential  enemy,  a  USV  is  sent  to  the  detection  area  to  identify  the  vehicle 
(i.e. ,  the  unidentified  vehicles  appear  as  dark  yellow  numbered  icons  in  map).  Engaging 
the  video  payload  that  arrives  at  a  detection  area  requires  the  operator  to  decide 
whether  the  vehicle  detected  is  a  potential  enemy.  If  an  enemy  is  identified,  a  UUV 
travels  to  the  location  to  target  the  enemy.  UUVs  are  slower  than  USVs  and  UAVs. 
UAVs  are  the  fastest  UVs. 
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Fig.  4.  RESCHU  SP  simulator  displays  a  mission  scenario  with  a  team  size  of  nine  UVs  (blue  numbered  icons  on  the 
map),  three  potential  enemies  (dark  yellow  numbered  icons  on  the  map),  and  one  identified  enemy  (red  numbered  icon  on 
the  map).  Notice  that  the  big  red  circle  is  the  asset  to  be  protected  (an  oil  platform,  while  the  big  yellow  circles  are  threat 
areas  that  the  UVs  should  avoid.  The  CSL  tab  shows  how  the  technology  handles  missions.  In  the  Active  section  of  the 
tab,  identify  and  attack  missions  that  are  currently  active  are  displayed  (i.e.,  box  E  displays  that  a  mission  to  identify 

potential  enemy  4  was  assigned  to  USV  9). 


iii.  Operator  Tasks 

The  operator’s  main  task  is  to  identify  and  target  potential  enemies  while 
protecting  an  asset  (i.e.,  oil  platform).  At  the  same  time,  the  operator  is  responsible  for 
supervising  the  path  of  the  UVs,  in  order  to  avoid  traveling  through  potential  threat 
areas  (bright  yellow  areas  on  the  map).  Threat  areas  are  zones  that  operators  should 
avoid  in  order  to  protect  the  health  of  their  vehicles.  Moreover,  operators  are  also 
responsible  for  following  chat  messages  which  provide  them  with  the  necessary 
Intelligence  and  guidance  to  complete  their  missions. 

When  a  UAV  detects  a  potential  enemy,  a  visual  flashing  alert  is  issued  to  warn 
the  operator.  This  alert  indicates  that  the  operator  should  command  the  CSL  system  to 
assign  a  UV  to  complete  the  task.  The  operator  commands  the  CSL  to  complete  the 
task  through  a  right-click  interaction.  The  CSL  system  chooses  a  UV  that  is  appropriate 
for  the  task  and  one  that  is  also  in  close  proximity  to  the  potential  target.  The  operator  is 
in  charge  of  approving  the  CSL  selection  by  submitting  the  task  through  the  Submit  All 
button  in  the  CSL  Editing  Controls  tab.  In  the  case  of  multiple  identification  tasks 
submitted  to  the  CSL  at  the  same  time,  the  operator’s  task  is  to  approve  the  CSL 
selection,  and  if  applicable,  determine  the  order  in  which  the  tasks  should  be  conducted. 
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For  example,  in  a  situation  in  which  there  is  only  one  UV  available  for  the  task,  the 
operator  has  to  determine  the  order  in  which  tasks  should  be  conducted  to  ensure  a 
good  mission  performance.  Once  the  order  of  tasks  has  been  determined,  the  operator 
needs  to  submit  the  commands  so  that  the  CSL  can  complete  the  tasks.  Once  that  a 
task  has  been  submitted,  a  selected  UV  is  sent  to  location,  when  it  arrives,  a  visual 
flashing  alert  warns  the  operator  that  the  video  payload  is  ready  to  engage.  Then,  the 
operator  engages  the  video  payload  through  a  right-click  interaction.  The  detected 
vehicle  is  viewed  through  the  video  image  displayed  in  the  Payload  View  tab  to 
determine  whether  the  detection  is  identified  as  the  enemy.  The  operator  identifies  the 
vehicle  by  clicking  on  the  Yes  or  No  button  below  the  payload  view.  A  supervisor  will 
inform  the  operator  via  chat  whether  the  identification  is  correct  or  not.  If  the  operator 
correctly  identifies  the  vehicle  as  an  enemy,  the  vehicle  icon  on  the  map  becomes  red.  If 
the  operator  incorrectly  identifies  a  detected  vehicle  as  the  enemy,  the  supervisor  will 
override  the  operator;  therefore,  the  icon  will  not  change  to  red.  The  next  step  for  the 
operator  is  to  inform  the  CSL  that  a  vehicle  should  be  assigned  to  complete  the  target 
mission.  Once  again,  the  CSL  system  chooses  a  UV  and  sends  it  to  the  target  location. 
When  on  target,  a  visual  flashing  alert  is  issued  to  inform  the  operator  that  the  UV  is 
ready  to  engage.  The  operator  confirms  this  through  a  right-click  interaction,  and  the 
target  is  eliminated.  In  this  way,  the  operator  is  responsible  to  identify  all  detections  and 
eliminate  all  enemies  in  order  to  protect  the  asset. 

iv.  Participants  and  Experimental  Procedure 

Experiments  are  being  conducted  using  the  RESCHU  SP  test  bed  in  order  to 
provide  data  for  model  validation.  The  experiment  was  designed  to  generate  a  large 
data  set  suitable  for  model  validation.  The  recruited  participants  are  students  from  the 
Naval  Postgraduate  School  (NPS).  The  online  test  bed  includes:  a  background  and  exit 
survey,  an  interactive  tutorial,  a  practice  trial,  and  one  of  a  set  of  possible  experimental 
conditions. 

The  objective  of  conducting  the  first  experiment  was  to  validate  the  model.  First, 
it  is  desired  to  have  performance  data  associated  with  the  different  levels  of  team  size, 
in  order  to  build  confidence  in  the  model’s  accuracy  at  replicating  human-UV-interaction 
under  different  conditions.  Second,  having  team  size  as  the  independent  variable,  the 
model’s  ability  to  replicate  statistically  significant  effects  on  the  operator  performance 
and/or  mission  performance  could  be  evaluated.  Finally,  having  data  sets  associated 
with  the  different  levels  of  team  size  allows  for  predictive  validation  by  selecting  a  single 
data  set  associated  with  one  of  the  conditions  and  predicting  the  results  observed  for  a 
second  condition. 

In  order  to  ensure  the  validity  of  the  variables  and  relationships  represented  in 
the  model,  the  decision  network  was  converted  into  a  Bayesian  Belief  Network  (BBN)  to 
run  validation  analysis.  The  software’s  Test  with  Cases  Analysis  is  used  to  validate  the 
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network.  The  analysis  examines  if  the  predictions  in  the  network  match  the  actual 
cases.  The  goal  of  the  test  is  to  divide  the  nodes  of  the  network  into  two  types  of  nodes: 
observed  and  unobserved.  The  observed  nodes  are  the  nodes  read  from  the  case  file, 
and  their  values  are  used  to  predict  the  unobserved  nodes  by  using  Bayesian  belief 
updating.  The  test  compares  the  predicted  values  for  the  unobserved  nodes  with  the 
actual  observed  values  in  the  case  file  and  the  successes  and  failures  are  then 
recorded.  The  report  produced  by  this  analysis  has  different  measures  that  validate 
each  node’s  predicted  capabilities.  After  evaluating  the  validity  of  the  model,  we  can 
determine  which  relationships  are  incorrect  and  we  can  make  the  network  learn  those 
relationships  through  the  collected  cases.  Finally,  we  can  run  sensitivity  analysis  and 
predictive  validation  analysis  to  determine  which  variable  has  the  biggest  effect  on  team 
size  and  how  each  variable  affects  the  overall  result  of  the  model. 

The  study  design  is  a  between-subject  design  with  three  conditions:  high  team 
size,  medium  team  size,  and  low  team  size.  The  high  team  size  condition  is  composed 
of  9  UVs:  3  UAVs,  3  USVs  and  3  UUVs.  The  medium  team  size  condition  is  composed 
of  7  UVs:  3  UAVs,  2  USVs  and  2  UUVs.  Finally,  the  low  team  size  condition  is 
composed  of  5  UVs:  3  UAVs,  1  USV  and  1  UUV.  Notice  that  the  UAV’s  number  was 
kept  constant  through  the  different  conditions  because  the  UAVs  produce  little 
interaction  with  the  operator  (i.e.,  UAVs  only  patrol  for  detection  and  operators  only 
have  to  supervise  their  flight  path  to  avoid  flying  into  threat  areas).  The  number  of  USVs 
and  UUVs  was  gradually  incremented  to  investigate  how  they  affect  the  performance 
measures  and  therefore  the  model  outcome.  Furthermore,  the  baseline  of  a  team  of  5 
UVs  was  decided  after  pilot  testing  the  simulation  with  different  team  sizes. 

The  experimental  test  bed  was  designed  for  a  web-based  delivery,  with  an 
interactive  tutorial  and  practice  trial.  A  web-based  experimentation  was  chosen  in  order 
to  obtain  as  much  data  as  possible.  The  website  was  Common  Access  Card  (CAC) 
protected  and  participation  was  via  invitation.  Data  collected  from  the  simulation  is 
currently  being  recorded  to  an  online  database.  Demographic  information  is  collected 
via  a  background  survey  presented  before  the  tutorial.  Participants  are  being  instructed 
to  maximize  their  overall  performance  by:  1)  avoiding  threat  areas  that  dynamically 
changed  and  therefore  minimizing  damage  to  the  UVs,  2)  correctly  identifying  enemies, 
3)  targeting  enemies  before  they  reach  the  asset,  4)  overriding  the  system  when 
necessary  to  minimize  vehicle  travel  times  and  maximize  mission  performance,  and  5) 
eliminating  potential  enemies  as  soon  as  possible.  Participants  have  to  go  through  the 
consent  form  and  background  survey  before  they  can  start  the  interactive  tutorial.  The 
estimated  time  to  read  through  the  consent  form  and  complete  the  background  survey  is 
10  minutes.  Next,  participants  are  allowed  to  go  through  the  interactive  tutorial  and 
practice  trial  until  they  feel  comfortable  with  the  task  and  the  interface.  The  estimated 
time  to  complete  these  two  sessions  is  about  25  minutes.  Once  they  are  comfortable, 
participants  can  move  to  the  test  session,  which  lasts  10  minutes.  After  completing  the 
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test  session,  participants  are  required  to  fill  out  an  exit  survey  that  last  about  5  minutes. 
Participants  are  allowed  to  see  their  test  score  at  the  end  of  the  test  session. 

v.  Experimental  Results 

Experiments  are  currently  being  conducted  at  NPS,  it  is  expected  that  the  data 
collection  will  last  until  the  end  of  May  201 1 . 
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❖  The  Department  of  Defense's  future  vision  for 

Network  Centric  Operations  (NCO)  is  intended  to 

increase  combat  control  by  networking  relevant 

entities  across  the  battlefield  [1]. 

v'  In  a  future  NCO  scenario,  one  operator  is  going  to 
supervise  a  team  of  heterogeneous  Unmanned  Vehicles 
(UVs)  to  work  together  towards  a  specific  task. 
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C2  Transition 
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Workload  Sources 
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Design  Challenges 


♦♦♦  Information  overload 

❖  Attention  allocation 

♦Mask  switching  and  interruption 

♦♦♦Trust  and  effectiveness  of  the  system  to  match 
mission  requirements 
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Motivation 


❖  To  successfully  develop  Unmanned  System's 
technology  for  NCO  scenarios  the  following  is 
required: 

S  Rigorous  mathematical  methods  and  tools  for  predicting 
the  behavior  of  newly  developed  C2  technologies  and  the 
operator  under  realistic  assumptions. 

s  Field  testing  approaches  to  identify  potential  problems 
and  prove  the  capabilities  and  robustness  of  the  new 
technologies. 
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Questions  of  Interest 


1.  With  the  current  overloaded  state  of  the 
information  of  most  C2  technologies,  how  can  we 
ensure  an  adequate  ratio  of  UVs  to  an  operator  in 
NCO  scenarios? 

2.  Are  new  C2  technologies  being  developed  for  NCO 
scenarios  truly  reducing  information  overload  and 
increasing  operator  performance  for  a  given  set  of 
mission  requirements? 
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Research  Goals 


♦♦♦  Develop  a  decision-making  tool  that  will  serve  to 
help  decision  makers: 

■S  Select  an  adequate  C2  technology  for  a  complex  mission 
scenario. 

•S  Determine  the  limitations  of  the  C2  technology  in  terms  of 
mission  requirements  (i.e.,  team  size,  performance  level). 

•S  Determine  areas  that  require  improved  designs  in  order  to 
modify  them  and  increase  operator  performance. 

■S  Develop  a  better  understanding  of  what  enables  operator 
capacity  in  this  type  of  mission  scenario. 
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Research  Approach 


❖  A  decision-making  tool  was  developed  using  a 
Decision  Network  (DN). 

v'  DN  uses  decision  theory,  which  is  a  close  cousin  of 
probability  theory,  to: 

o  Specify  the  desirability  of  various  outcomes  and  the  cost  of  various 
actions  that  may  be  performed  to  affect  the  outcome. 

o  Find  an  action  or  plan  that  maximizes  the  expected  utility  minus 
the  cost. 
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Modeling  Operator  Capacity 


•  Information 
Processing 
theories  [7]: 

•  What  affects 
performance? 


What  affect  the  operator  in 
complex  tasks? 


•  Simplified 
version  [4]: 

•Only  1  UV 
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Modified  Version 

M5]: 

•Multiple 
homogeneous 
UVs 


Complex  Mission  Task 


Operator  Co^ihitive 
Capacity  Modi 


•  NCO  Scenario:  multiple  heterogeneous 
UVs,  high  workload,  simultaneous  tasks, 
multiple  enemies,  etc. 


How  is  the  situation 
modeled? 


I 


•  Model  the 
situation  [6]: 

•Discrete  event 
simulation. 
•Heterogeneous 
UVs. 

•UT  as  workload 
measure. 
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Model  Details 


Sequence  of  DM  Process  Input:  Observable  Measures  Output:  Unobservable 
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Experimental  Methodology 


X 

ivelop  simulati 
ivironment  wil 
potential  NCO 

on  ^ 

th 

Validate  Model/  Learn  from  data: 

Are  the  relationships  the  same?  If 
not,  what  are  the  correct 
relationships  observed? 


L 


Can  the  model  predict  whether  the 
user  can  control  more  or  less  UVs? 


Develop  workload  and  SA  curves 
that  will  help  us  determine  the 
operator's  limitations. 
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Experimental  Trials 


♦♦♦  Experimental  trials  are  currently  being  conducted 
online  at  Naval  Postgraduate  School  (NPS). 

•S The  online  trial  includes:  a  background  and  exit  survey,  a 
tutorial,  a  trial  game,  and  one  of  a  set  of  possible 
experimental  conditions. 

❖  The  study  design  will  be  a  between  subjects  with 
three  conditions: 

■S  High  team  size  (9  UVs),  medium  team  size  (7  UVs),  and  low 
team  size  (5  UVs). 


13 


SPAWAR 

_  _  ® 

▼ 

Systems  Center 
PACIFIC 


Research  Implications 


❖  Better  understanding  of  operator  cognitive  capacity 
in  complex  UV  mission  scenarios. 

❖  Development  of  specific  C2  design  requirements  for 
complex  UV  environments. 

❖  Definition  of  the  performance  measures  for  this  type 
of  complex  mission  scenario. 
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Future  Research 


♦♦♦Model  Validation  Phase  2: 

^Validate  the  model  with  a  different  NCO 
technology. 

V Incorporate  workload  and  SA  curves  into  the 
model  to  strength  prediction. 

V Start  working  on  a  decision  tool  package  that  will 
include  model  and  program  to  extract  input  from 
any  simulation. 

♦♦♦Investigate  the  development  of  a  warning 
tool  based  on  our  decision  net. 
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Questions? 
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